Standardized ILR-Based and Task-Based Speech-to-Speech MT Evaluation

We present a new method for task-based speechto-speech machine translation evaluation, in which tasks are defined and assessed according to independent published standards, both for the military tasks performed and for the foreign language skill levels used. We analyze task success rates and automatic MT evaluation scores for 220 role-play dialogs. Each role-play team consisted of one native English-speaking soldier role player, one native Pashto-speaking local national role player, and one Pashto/English interpreter. Machine translation (MT) and human translation (HT) conditions were assigned in a Latin Square design. Dialogs were assessed for language difficulty according to the Interagency Language Roundtable (ILR) speaking and listening skills.